Adaptive Regularization Method Research Articles

Improved adaptive regularization for simulated annealing inversion of transient electromagnetic

Geophysical inversion usually involves ill-posed problem. Regularization is the most commonly used method to mitigate this problem. There are many regularization parameter selection methods, among which the adaptive regularization method can automatically update parameters during iteration, reducing the difficulty of parameter selection. Therefore, it is widely used in linear inversion. However, there are very few studies on the use of adaptive regularization methods in stochastic optimization algorithms. The biggest difficulty is that in stochastic optimization algorithms, the search direction of any iteration is completely random. Data fitting term and stabilizing term vary in a wide range, making it difficult for traditional methods to work. In this paper, we consider the contributions of the data fitting term and the stabilizing term in the objective function and give an improved adaptive regularization method for very fast simulated annealing (VFSA) inversion for transient electromagnetic (TEM) data. The optimized method adjusts the two terms dynamically to make them in balance. We have designed several numerical experiments, and the experimental results demonstrate that the method in this paper not only accelerates the convergence, but also the inversion results are very little affected by the initial regularization parameter. Finally, we apply this method to field data, and the inversion results show very good agreements with nearby borehole data.

Relaxation linear regression with spectral–spatial constrained locality adaptive regularization for hyperspectral image classification

AbstractRecently, relaxation linear regression has received increasing attention in image analysis. However, the current relaxation linear regression methods fail to consider the local geometrical structure and spatial information while they are applied for hyperspectral image (HSI) classification. To address the above problems, this letter proposes a novel relaxation linear regression with spectral–spatial constrained locality adaptive regularization (SSLA‐RLR) method for HSI classification. The SSLA‐RLR method not only integrates the locality adaptive graph with relaxation linear regression to adaptively exploit the local geometrical structure for relieving the side effects of noise corruptions, but also takes the spatial correlations between each data point and its neighbours into consideration via spectral–spatial constraint. Finally, extensive experiments are conducted on three benchmark HSI datasets to demonstrate the effectiveness of the proposed method.

Enhancing Alzheimer Disease Segmentation through Adaptively Regularized Weighted Kernel-Based Clustering

Image segmentation is important in image analysis because it helps to locate objects and boundaries within a picture. This study offers Adaptively Regularized Weighted Kernel-Based Clustering (ARWKC), a unique segmentation technique built exclusively for recovering brain tissue from medical pictures. The proposed approach incorporates adaptive regularization and weighted kernel-based clustering techniques to increase the accuracy and resilience of brain tissue segmentation. The picture is initially preprocessed with the ARWKC method to improve its quality and eliminate any noise or artifacts. The adaptive regularization method is then utilized to effectively deal with the visual variation of brain tissue in clinical images. This adaptive regularization contributes to more accurate and consistent segmentation outcomes. The weighted kernel-based clustering method is then used to find and group pixels with comparable properties, with a focus on brain tissue areas. This clustering approach employs a weighted kernel function that takes into account both geographical closeness and pixel intensities, allowing the algorithm to capture local picture features and improve segmentation accuracy. Extensive experiments were conducted on a collection of medical images to evaluate the efficacy of the ARWKC algorithm. The well-known k-means clustering method, often used in image segmentation applications, was utilized as a benchmark for comparison. In terms of accuracy and resilience for brain tissue segmentation, the experimental findings showed that the ARWKC method surpasses the k-means clustering approach.

Scalable adaptive cubic regularization methods

Scalable adaptive cubic regularization methods

Adaptive anadromic regularization method for the Cauchy problem of the Helmholtz equation

In this research work, we investigate the Cauchy problem for the Helmholtz equation. Considering the completion data problem in a bounded cylindrical domain with Neumann and Dirichlet conditions on a part of the boundary. An immediate approximation of missing boundary data is obtained using a method that factorizes the boundary value problem. This factorization uses the Neumann to Dirichlet or Dirichlet to Neumann operators that satisfy the Riccati equation. Some singularities appear in the solution of the Riccati equation for a particular length of the waveguide of the Helmholtz equation. We elaborate a new numerical method called ‘adaptive anadromic regularization method’ that can solve these operators beyond the singularity. In addition, we introduce a scaling matrix technique to the linear matrix equations associated with the Riccati equations to generate normalized solutions. Our numerical procedure not only approximates missing boundary data, but also provides an error estimate that allows efficient time stepping. Numerical tests proved to confirm the theory even in the presence of high noise levels.

An adaptive regularization method in Banach spaces

This paper considers optimization of nonconvex functionals in smooth infinite dimensional spaces. It is first proved that functionals in a class containing multivariate polynomials augmented with a sufficiently smooth regularization can be minimized by a simple linesearch-based algorithm. Sufficient smoothness depends on gradients satisfying a novel two-terms generalized Lipschitz condition. A first-order adaptive regularization method applicable to functionals with β-Hölder continuous derivatives is then proposed, that uses the linesearch approach to compute a suitable trial step. It is shown to find an ϵ-approximate first-order point in at most evaluations of the functional and its first p derivatives.

Three-dimensional inhomogeneous temperature tomography of confined-space flame coupled with wall radiation effect by instantaneous light field

Optical tomography has been demonstrated to be a powerful tool for the three-dimensional (3D) measurement of the dynamic characteristics of turbulent flame. Many practical visualizations of reaction flows require imaging through optical windows on the combustor, which have complex interactions with wall temperature, reflection characteristics, and limited optical access, which may seriously influence the imaging quality. Previous work focused on unconfined flame imaging, and lacked the study of the multiple reflection effects of the wall and the tomographic reconstruction within a wide temperature range. To solve these problems, the effect of chamber wall radiation on the photothermal information is analyzed in the forward problem, and the flame temperature with wall radiation is reconstructed in the inverse problem. In the forward problem, the light field acquisition model of spontaneous emission of the confined flame is developed through the backward Monte Carlo method based on the radiation distribution factor, which can comprehensively consider the essential effects such as radiative attenuation, wall reflection, medium scattering interference, and limited optical access. The recorded light field signals indicate that the reflection pattern and transfer mechanism of the wall material could affect the contribution of radiative source terms in different discrete domains to the detected energy. A novel tomography algorithm, namely, the adaptive threshold segmentation iterative regularization (ATSIR) method, has been proposed to extend the temperature range of tomography. The key of this method is to carry out a two-step reconstruction of the high and low-temperature regions through the temperature threshold, and introduce the prior smoothing information to alleviate the ill-posedness of tomography. The visualization results show that the well-established algorithms (e.g., least-square QR-factorization and regularization) have lower reconstruction quality for the combustion field within a wide temperature range, even if the regularization term is added. However, the proposed ATSIR method has the advantages of better suppression of fluctuations, smaller reconstruction error, and stronger anti-noise ability. The detailed analysis of the radiative transfer mode and 3D temperature visualization of the combustion field in a confined space can provide a valuable guideline for the safety design of the engine combustion chamber and fuel utilization.

Adaptive graph regularization method based on least square regression for clustering

The low-rank representation with adaptive graph regularization (LRAGR) method has been successfully proposed for clustering applications, since it can solve the problems of missing local information and unclear graph structure in data clustering tasks. However, LRAGR utilizes the low-rank representation technique to mine data information, which makes the obtained coefficient matrix too dense and is not conducive to cluster division. Moreover, due to the singular value decomposition, the constraint of the coefficient matrix requires high computational complexity and is difficult to be applied in practical tasks. To address the above issues, in this paper, a new adaptive graph regularization method, called the adaptive graph regularization method based on least squares regression (LSAGR), is proposed for data clustering applications. Specifically, the proposed LSAGR method adopts the Frobenius norm instead of kernel norm to approximate the rank function to satisfy the clustering effect, that is, the coefficients of the cluster-related data are approximately equal. Compared with the traditional LRAGR method, in clustering tasks, the LSAGR method can better reveal the real subspace membership, improve the clustering performance and reduce the computational complexity. Finally, extensive experimental results demonstrate that, compared with several related state-of-the-art methods, the proposed LSAGR method usually has better clustering performance on six real-world image datasets in clustering applications.

An Adaptive Sparse Regularization Method for Response Covariance-Based Structural Damage Detection

Structural damage detection is usually an ill-posed inverse problem due to the contamination of measurement noise and model error in structural health monitoring. To deal with the ill-posed damage detection problem, l2-regularization is widely used. However, l2-regularization tends to provide nonsparse solutions and distribute identified damage to many undamaged elements, potentially leading to false alarms. Therefore, an adaptive sparse regularization method is proposed, which considers spatially sparse damage as a prior constraint since structural damage often occurs in some locations with stiffness reduction at the sparse elements out of the large total number of elements in an entire structure. First, a response covariance-based convex cost function is established by incorporating an l1-regularized term and an adaptive regularization factor to formulate the sparse regularization-based damage detection problem. Then, optimal sensor placement is conducted to determine the optimal measurement locations where the acceleration responses are adopted for computing the response covariance-based damage index and cost function. Further, the predictor-corrector primal-dual path-following approach, an efficient and robust convex optimization algorithm, is applied to search for solutions to the damage detection problem. Finally, a comparison study with the Tikhonov regularization-based damage detection method is conducted to examine the performance of the proposed adaptive sparse regularization-based method by using an overhanging beam model subjected to different damage scenarios and noise levels. The numerical study demonstrates that the proposed method can effectively and accurately identify damage under multiple damage scenarios with various noise levels, and it outperforms the Tikhonov regularization-based method in terms of high accuracy and few false alarms. The analyses on time consumption, adaptiveness of the sparse regularization factor, model-error resistance, and sensor number influence are conducted for further discussions of the proposed method.

Noise-aware clustering based on maximum correntropy criterion and adaptive graph regularization

Graph-based clustering is a basic subject in the field of machine learning, but most of them still have the following deficiencies. First, similarity graph construction and data division into corresponding classes are always divided into two independent steps. Second, noise contained in real data may cause the learned similarity graph to be inaccurate. Third, the traditional metrics based on Euclidean distance is difficult to tackle non-Gaussian noise. In order to eliminate these limitations, a noise-aware clustering based on correntropy and adaptive graph regularization method (NCCAGR) is proposed. 1) In order to change the problem from two-steps to single-step, we formulate a joint clustering learning framework that simultaneously learns a robust similarity graph and performs data clustering; 2) To overcome the influence of noise, we construct a Laplacian matrix and perform adaptive graph regularization based on clean data; 3) By introducing the correntropy to solve the problem of non-Gaussian noise and heavy tail in the original data. Furthermore, a half-quadratic optimization method is used to transform the problem into a quadratic form to facilitate subsequent solutions. Finally, experiments show that the proposed method not only has high performance, but also outperforms both classical methods and state-of-the-art methods in robustness.

A Bayesian Framework of Non-Synchronous Measurements at Coprime Positions for Sound Source Localization With High Resolution

The noise distribution in the mechanical system is tightly connected to the structure design and particular operating conditions. The noise source localization can effectively assist with the operating condition monitoring and noise reduction design of the mechanical device. The performance limitation of the array aperture for lower frequency acoustic localization is broken up by the non-synchronous measurement at coprime positions (CP-NSM), while this method has high uncertainty and requires an efficient adaptive regularization method to solve its corresponding acoustic inverse problem. The algorithms under the Bayesian framework with Student-t priors (variational Bayesian approximation and subspace variational Bayesian) are derived and deployed to solve the acoustic inverse problem in the CP-NSM method, and the results are compared with those obtained by the interior-point method and the alternating direction method of the multipliers algorithm. The proposed Bayesian methods have the advantages of adaptive regularization parameter estimation, which can reduce the influence of various interferences in CP-NSM. At the same time, in addition to using simulations and experiments in the anechoic chambers, the proposed Bayesian algorithms are validated further in real industrial applications. The proposed method is applied to improve the noise reduction design of the centrifugal fan, a mechanical device containing more information from noise sources.

OFFO minimization algorithms for second-order optimality and their complexity

An Adagrad-inspired class of algorithms for smooth unconstrained optimization is presented in which the objective function is never evaluated and yet the gradient norms decrease at least as fast as \(\mathcal{O}(1/\sqrt{k+1})\) while second-order optimality measures converge to zero at least as fast as \(\mathcal{O}(1/(k+1)^{1/3})\). This latter rate of convergence is shown to be essentially sharp and is identical to that known for more standard algorithms (like trust-region or adaptive-regularization methods) using both function and derivatives’ evaluations. A related “divergent stepsize” method is also described, whose essentially sharp rate of convergence is slighly inferior. It is finally discussed how to obtain weaker second-order optimality guarantees at a (much) reduced computational cost.

Adaptive regularization minimization algorithms with nonsmooth norms

AbstractAn adaptive regularization algorithm (AR$1p$GN) for unconstrained nonlinear minimization is considered, which uses a model consisting of a Taylor expansion of arbitrary degree and regularization term involving a possibly nonsmooth norm. It is shown that the nonsmoothness of the norm does not affect the ${\mathcal {O}}(\epsilon _1^{-(p+1)/p})$ upper bound on evaluation complexity for finding first-order $\epsilon _1$-approximate minimizers using $p$ derivatives, and that this result does not hinge on the equivalence of norms in $\mathbb {R}^n$. It is also shown that, if $p=2$, the bound of ${\mathcal {O}}(\epsilon _2^{-3})$ evaluations for finding second-order $\epsilon _2$-approximate minimizers still holds for a variant of AR$1p$GN named AR2GN, despite the possibly nonsmooth nature of the regularization term. Moreover, the adaptation of the existing theory for handling the nonsmoothness results is an interesting modification of the subproblem termination rules, leading to an even more compact complexity analysis. In particular, it is shown when the Newton’s step is acceptable for an adaptive regularization method. The approximate minimization of quadratic polynomials regularized with nonsmooth norms is then discussed, and a new approximate second-order necessary optimality condition is derived for this case. A specialized algorithm is then proposed to enforce first- and second-order conditions that are strong enough to ensure the existence of a suitable step in AR$1p$GN (when $p=2$) and in AR2GN, and its iteration complexity is analyzed. A final section discusses how practical approximate curvature measures may lead to weaker second-order optimality guarantees.

Deep Learning Model With Adaptive Regularization for EEG-Based Emotion Recognition Using Temporal and Frequency Features

Since EEG signal acquisition is non-invasive and portable, it is convenient to be used for different applications. Recognizing emotions based on Brain-Computer Interface (BCI) is an important active BCI paradigm for recognizing the inner state of persons. There are extensive studies about emotion recognition, most of which heavily rely on staged complex handcrafted EEG feature extraction and classifier design. In this paper, we propose a hybrid multi-input deep model with convolution neural networks (CNNs) and bidirectional Long Short-term Memory (Bi-LSTM). CNNs extract time-invariant features from raw EEG data, and Bi-LSTM allows long-range lateral interactions between features. First, we propose a novel hybrid multi-input deep learning approach for emotion recognition from raw EEG signals. Second, in the first layers, we use two CNNs with small and large filter sizes to extract temporal and frequency features from each raw EEG epoch of 62-channel 2-s and merge with differential entropy of EEG band. Third, we apply the adaptive regularization method over each parallel CNN’s layer to consider the spatial information of EEG acquisition electrodes. The proposed method is evaluated on two public datasets, SEED and DEAP. Our results show that our technique can significantly improve the accuracy in comparison with the baseline where no adaptive regularization techniques are used.

A multiparametric volumetric quantitative ultrasound imaging technique for soft tissue characterization.

Quantitative ultrasound (QUS) offers a non-invasive and objective way to quantify tissue health. We recently presented a spatially adaptive regularization method for reconstruction of a single QUS parameter, limited to a two dimensional region. That proof-of-concept study showed that regularization using homogeneity prior improves the fundamental precision-resolution trade-off in QUS estimation. Based on the weighted regularization scheme, we now present a multiparametric 3D weighted QUS (3D QUS) method, involving the reconstruction of three QUS parameters: attenuation coefficient estimate (ACE), integrated backscatter coefficient (IBC) and effective scatterer diameter (ESD). With the phantom studies, we demonstrate that our proposed method accurately reconstructs QUS parameters, resulting in high reconstruction contrast and therefore improved diagnostic utility. Additionally, the proposed method offers the ability to analyze the spatial distribution of QUS parameters in 3D, which allows for superior tissue characterization. We apply a three-dimensional total variation regularization method for the volumetric QUS reconstruction. The 3D regularization involving N planes results in a high QUS estimation precision, with an improvement of standard deviation over the theoretical 1/N rate achievable by compounding N independent realizations. In the in vivo liver study, we demonstrate the advantage of adopting a multiparametric approach over the single parametric counterpart, where a simple quadratic discriminant classifier using feature combination of three QUS parameters was able to attain a perfect classification performance to distinguish between normal and fatty liver cases.

Joint seismic and petrophysical nonlinear inversion with Gaussian mixture-based adaptive regularization

The estimation of reservoir parameters from seismic observations is one of the main objectives in reservoir characterization. However, the forward model relating the petrophysical properties of rocks to observed seismic data is highly nonlinear, and solving the relevant inverse problem is a challenging task. We have developed a novel inversion method for jointly estimating the elastic and petrophysical parameters of rocks from prestack seismic data. We combine a full rock-physics model and the exact Zoeppritz equation as the forward model. To overcome the ill conditioning of the inverse problem and address the complex prior distribution of model parameters given lithofacies variations, we have developed a regularization term based on the prior Gaussian mixture model under a Bayesian framework. The objective function is optimized by the fast simulated annealing algorithm, during which the Gaussian mixture-based regularization terms are adaptively and iteratively adjusted by the maximum likelihood estimator, allowing the posterior distribution to be more consistent with the observed seismic data. The adaptive regularization method improves the accuracy of petrophysical parameters compared with sequential inversion and nonadaptive regularization methods, and the inversion result can be used for indicating gas-saturated areas when applied to field data.

Ensemble clustering-based fault diagnosis method incorporating traditional and deep representation features

The traditional and deep representation features have been successfully employed in the clustering-based fault diagnosis, however, current studies have ignored their heterogeneity and complementarity. Besides, these studies have rarely explored the superiority of ensemble clustering, leading to unstable diagnostic results. To address these issues, a novel ensemble clustering-based fault diagnosis method, namely ECFD, is proposed based on the collaboration of traditional and deep representation features. To the best of our knowledge, this is the first attempt to develop an ensemble clustering method with these types of features for fault diagnosis of rotating machinery. Initially, features are extracted by different approaches, among which the traditional features are obtained by time-domain, frequency-domain, time–frequency-domain approaches, and the deep representation features are extracted by the gated recurrent units-based autoencoders. Then, the heterogeneous and complementary information among these features is explored by the adaptive group regularization method, which is also developed to generate high-quality feature subspaces for constructing the base learners of the ensemble clustering. Finally, the clustering results of each base learner are integrated by a weighted voting method. The effectiveness of the proposed ECFD was verified by the experiments on the real-world bearing vibration dataset. The results indicate that the proposed ECFD can explore the heterogeneity and complementarity among different kinds of features and take advantage of the ensemble clustering, which is superior to other existing benchmark clustering methods in unsupervised fault diagnosis.

Accurate AM-FM signal demodulation and separation using nonparametric regularization method

In this paper we propose a novel adaptive nonparametric regularization (NPR) method for solving optimization problems with linear constraint. The regular parameter in NPR algorithm is adaptively updated. We prove that the NPR is convergent and provide an early stop method (ESM) with a termination criterion to handle the perturbation problem which is unsolved in other methods such as the augmented Lagrange method (ALM). We then introduce a new differential operator which can absolutely annihilate the amplitude-modulated and frequency-modulated (AM-FM) signals (AM-FM operator, AFO). The proposed operator is a precise operator and thus can obtain a more accurate solution in operator based signal demodulation and separation problems. We apply the NPR algorithm in signal demodulation and separation based on the AFO and propose signal demodulation (NPR-AFOSD) and separation (NPR-AFOSS) algorithms. The experimental results on both synthetic AM-FM signals and the real-life data demonstrate that the proposed demodulation and separation methods are more effective and robust than the state-of-the-art methods.

An adaptive non-convex hybrid total variation regularization method for image reconstruction in electrical impedance tomography

The image reconstruction of conductivity distribution in electrical impedance tomography (EIT) is a seriously ill-posed inverse problem. To cope with the problem, it is recognized that the regularization method is an effective approach. In this paper, an adaptive non-convex hybrid total variation (ANHTV) regularization method is proposed to reconstruct the conductivity distribution in EIT. The iterative reweighted least squares algorithm and the iterative alternating direction method of multipliers algorithm are developed to solve the ANHTV-based inverse model in the image reconstruction. Besides, all the parameters utilized in the inverse model are adaptively selected. To validate the advantage of the proposed method, extensive numerical simulation and experimental work have been carried out. Also, qualitative and quantitative comparisons with two convex TV-based regularization methods are conducted. The results show that the proposed method is more advantageous in terms of staircase effect suppression, edge information preservation and noise resisting in the image reconstruction.

An accelerated first-order method with complexity analysis for solving cubic regularization subproblems

We propose a first-order method to solve the cubic regularization subproblem (CRS) based on a novel reformulation. The reformulation is a constrained convex optimization problem whose feasible region admits an easily computable projection. Our reformulation requires computing the minimum eigenvalue of the Hessian. To avoid the expensive computation of the exact minimum eigenvalue, we develop a surrogate problem to the reformulation where the exact minimum eigenvalue is replaced with an approximate one. We then apply first-order methods such as the Nesterov’s accelerated projected gradient method (APG) and projected Barzilai-Borwein method to solve the surrogate problem. As our main theoretical contribution, we show that when an $$\epsilon$$ -approximate minimum eigenvalue is computed by the Lanczos method and the surrogate problem is approximately solved by APG, our approach returns an $$\epsilon$$ -approximate solution to CRS in $${\tilde{O}}(\epsilon ^{-1/2})$$ matrix-vector multiplications (where $${\tilde{O}}(\cdot )$$ hides the logarithmic factors). Numerical experiments show that our methods are comparable to and outperform the Krylov subspace method in the easy and hard cases, respectively. We further implement our methods as subproblem solvers of adaptive cubic regularization methods, and numerical results show that our algorithms are comparable to the state-of-the-art algorithms.